Comparing χ tests for separability

This paper describes a series of novel statistical meta-tests for comparing experimental runs for significant difference, for conditions where experiments are carried out using Binomial or multinomial contingency statistics (χ, z, log-likelihood tests, etc.). The new tests permit us to evaluate whether experiments have failed to replicate on new data; whether a particular data source or subcorpus obtains a significantly different result than another; or whether changing experimental parameters obtains a stronger effect. Recognising when an experiment obtains a significantly different result and when it does not is an issue frequently overlooked in research publication. Papers are frequently published citing ‘p values’ or test scores suggesting a ‘stronger effect’, substituting for sound statistical reasoning. This paper sets out a series of tests which together illustrate the correct approach to this question, namely, to compute confidence intervals for differences between effect sizes and distributions. The meta-tests presented are derived mathematically from the χ test and the Wilson score interval, and cover both ‘goodness of fit’ and pairwise ‘homogeneity’ tests. Meta-tests for comparing tests with one degree of freedom (‘2 × 1’ and ‘2 × 2’ tests) are generalised to those of arbitrary size (‘r × 1’ and ‘r × c’). Finally, we compare our approach with a competing approach offered by Zar (1999), which, while straightforward to calculate, turns out to be both less powerful and less robust.